The present subject matter relates to a method of, and system for, providing feedback on source code being created. In one embodiment, the invention may be used for real-time test case pattern matching.
Software testing is an investigation carried out to provide information about the quality of the software product under test, in relation to the environment in which it is intended to operate. This includes, for example, the execution of a program with the intent of finding errors. The primary purpose of such software testing is to discover failures so that the detected errors may be corrected. The scope of software testing often includes the execution of the code in various environments and under different conditions. In the current culture of software development, a testing organization may be separate from the development team. Information derived from the software testing may be used to correct the original process by which the software is developed.
With the increasing time and resource pressures on test organizations it is vitally important that every new test case that is written is as effective as possible. Test effectiveness, in this context, means that a new test case would exercise the maximum amount of the system under test's (SUT) untested code with the minimum amount of overlap and duplication of existing test cases. The ideal situation in relation to testing using a test suite would be to have test cases that all exercise different areas of the software code. The main benefit being that there is no duplication of test code or effort. However, there is an added benefit in that it becomes easier to isolate bugs.
In the case where each test is testing a different part of the code it is likely that one bug in the SUT may only cause one test case to fail. In reality, the coverage provided by the tests tends to be creation tests that are overlapping, in relation to the code that comprises the SUT. This results in large amounts of duplication and overlap and consequently, large amounts of wasted effort. It also means that a single bug may be likely to cause the failure of multiple test cases, thus making debugging more complex. The problem is that the test case programmer receives little help in determining the effectiveness of their test until it is too late to do anything about it.
The current solution to this problem is to use code coverage instrumentation, for example, tools such as EMMA (emma.sourceforge.net) during the test execution phase, and then carry out a complex analysis of the output to determine which SUT code is covered by each test case. At this point, overlaps and duplication may be identified so that a more modular test suite may be produced on the second pass. However, the main drawback to this solution is that it is necessary to write all the test code, set up the SUT, instrument the SUT, execute the test cases, analyse the code coverage output, and finally make the assessment on each test's effectiveness before the test programmer may take any action. This is a very time consuming and costly exercise to do properly. In addition, the test code programmers are very reluctant to change what is probably already a valid test case just to remove the duplication. So, valuable time and resources may be wasted.